fix: skip metadata-only SSE chunks instead of aborting stream in StreamingResponseAggregator#783
fix: skip metadata-only SSE chunks instead of aborting stream in StreamingResponseAggregator#783anishesg wants to merge 1 commit into
Conversation
…amingResponseAggregator ## Link to Issue or Description of Change Signed-off-by: anish k <ak8686@princeton.edu>
kdroste-google
left a comment
There was a problem hiding this comment.
@anishesg, thank you for your contribution!
The fix seems OK.
Please see other comments for minor remarks
Thank you!
| if len(genResp.Candidates) == 0 { | ||
| // shouldn't happen? | ||
| yield(nil, fmt.Errorf("empty response")) | ||
| // Vertex AI occasionally emits leading SSE chunks that contain only |
There was a problem hiding this comment.
Please add the information that sometimes SSE stream can start with an empty text parts mixed with chunks which do not contain any candidates.
| } | ||
| } | ||
|
|
||
| func TestMetadataOnlyChunkDoesNotAbortStream(t *testing.T) { |
There was a problem hiding this comment.
Please add a test for the following sequence
Empty text
No candidates
No candidates
Empty text
No candidates
Non-empty text
|
I'm closing this PR, the #918 is a replacement |
|
No longer applicable |
|
@anishesg , thank you! |
|
Thanks for jumping on this so quickly, @anishesg — the diagnosis here was We ended up merging #918, which builds directly on your fix. The one change: The fix shipped in v1.4.0. Thanks again for the contribution. |
Link to Issue or Description of Change
Closes: #782
Vertex AI occasionally emits a leading SSE chunk that contains only response metadata (
createTime,modelVersion,usageMetadata, etc.) with noCandidatesfield. This happens consistently withgemini-3-flash-preview+googleSearchgrounding and caused roughly 40–50% of streaming calls to fail with"empty response", even though the actual content arrived in subsequent chunks.The root cause is in
internal/llminternal/stream_aggregator.goinProcessResponse: whenlen(genResp.Candidates) == 0, the function calledyield(nil, fmt.Errorf("empty response"))and returned, aborting the iterator. The fix changes this to a silentreturn, allowing the aggregator to skip metadata-only chunks and continue processing subsequent chunks that carry real content. This matches the behavior ofadk-python'sPROGRESSIVE_SSE_STREAMINGpath which handles identical chunks gracefully.The unused
fmtimport was also removed as a consequence.Testing Plan
Unit Tests
TestMetadataOnlyChunkDoesNotAbortStreaminstream_aggregator_test.gothat feeds a metadata-only chunk (noCandidates) followed by a real content chunk and asserts the aggregated response contains the expected text without any error.Manual E2E Tests
The reproduction script from the issue (10 runs of
gemini-3-flash-preview+googleSearchwithStreamingModeSSE) now produces 10/10 successes with this patch applied, matching the behavior ofStreamingModeNoneandadk-python.Checklist
Fixes #782